The single-channel image super-resolution method can not achieve both fast convergence and high quality texture detail restoration. In order to solve the problem, a new Enhanced algorithm of image Super-Resolution based on Dual-channel Convolutional neural network (EDCSR) was proposed. Firstly, the network was divided into deep channel and shallow channel. Deep channel was used to extract detailed texture information of images, and shallow channel was mainly used to restore the overall contour of images. Then, the advantages of residual learning were used by the deep channel to deepen network and reduce parameters of model, eliminate the network degradation problem caused by too deep network. The long and short-term memory blocks were constructed to eliminate the artifacts and noise caused by the deconvolution layer. The texture information of image at different scales were extracted by a multi-scale method, while the shallow channel only needed to be responsible for restoring the main contour of image. Finally, the dual-channel losses were integrated to optimize the network continuously, which guided the network to generate high-resolution images. The experimental results show that, compared with the End-to-End image super-resolution algorithm via Deep and Shallow convolutional networks (EEDS), the proposed algorithm converges more quickly, image edge and texture reconstruction effects are significantly improved, the Peak Signal-to-Noise Ratio (PSNR) and Structural SIMilarity (SSIM) are improved averagely by 0.15 dB and 0.0031 on data set Set5, while these are improved averagely by 0.18 dB and 0.0035 on data set Set14.